Testing the incremental effectiveness of pay-for-performance to improve implementation of a motivational interviewing brief intervention for substance use disorders in HIV settings: Results of a cluster-randomized type 3 hybrid trial

Background: Substance use disorders (SUDs) have a serious adverse impact on people living with HIV. Previously, using a 39-site dual-randomized type 2 hybrid trial design, findings from the Substance Abuse Treatment to HIV Care Project supported the Implementation and Sustainment Facilitation (ISF) strategy to improve implementation and effectiveness of a motivational interviewing brief intervention (MIBI) for SUD within HIV service settings across the United States (US). Building on this trial, this cluster-randomized type 3 hybrid trial aimed to test the incremental effectiveness of a pay-for-performance (P4P), a form of the “alter incentive/allowance structures” strategy. Methods: Twenty-six HIV service organizations, their staff participants (N=87), and their client participants (N=341) were cluster-randomized to one of two implementation conditions. The control condition included staff-focused training, feedback, and consultation (TFC) and team-focused implementation and sustainment (ISF). The experimental condition included TFC+ISF as well as P4P (TFC+ISF+P4P). P4P used financial incentives to reward MIBI implementation (US$10 per MIBI delivered) and MIBI implementation at or above a pre-defined level of quality (US$10 per demonstration). In addition to these outcomes, past 4-week changes/reductions in client participant’s days of primary substance use and anxiety symptoms were examined. Results: The addition of P4P had a large and significant effect on the number of MIBIs implemented (d=1.30, p<.05) and reduction in anxiety (d=−1.54), but there was no impact on days of substance use. P4P had large effects on MIBI quality (d=1.24) and MIBI implementation effectiveness (d=1.28), but these were not significant (p<.10). Conclusions: P4P is a form of the “alter incentive/allowance structures” strategy Its function is to reward the implementation of a clinical innovation. Rewarding implementation is consistent with the theory of implementation effectiveness, which suggests implementation climate (i.e., the extent to which implementation is expected, supported, and rewarded) is a key antecedent of implementation effectiveness (i.e., the consistency and quality of implementation). We found that P4P had a significant, positive impact on MIBI implementation in HIV service settings, but client-level outcomes were mixed. Future research should examine the cost-effectiveness of this strategy, as well as to examine the effectiveness of P4P to improve the implementation of other evidence-based innovations. Trial registration: ClinicalTrials.gov: NCT04687917. Registered 12/18/2020.


Background
The United States' Ending the HIV Epidemic initiative, announced in early 2019, put forth an ambitious target of achieving 90% reduction in new HIV infections within 10 years [1].Substance use disorders (SUDs) continue to pose a barrier to this goal [refs].For example, in May of 2019, just a few months after the announcement of this initiative and before the COVID-19 pandemic began, Garner and colleagues (2022) [2] conducted a national survey to estimate the prevalence and negative impacts of ve different SUDs among people with HIV (PWH) in the United States.Estimates varied by region of the United States, but overall, average prevalence rates were 42.3% for cannabis, 41.9% for alcohol, 34.6% for opioid, 32.2% for methamphetamine, and 28.1% for cocaine.In terms of the negative impacts that each SUD has on HIV care (e.g., being linked to HIV care, being retained in HIV care) and (e.g., having stable housing, being employed), it was estimated the most problematic was methamphetamine and the least problematic was cannabis.When the prevalence rate and negative impact estimates were multiplied to estimate the population-level negative impact, the top three most problematic use disorders were alcohol, methamphetamine, and opioids.Because COVID-19 has been reported to have increased the use of alcohol and other substances as well as the prevalence of anxiety [3][4][5][6], the rates of comorbid HIV, SUD and mental health problems may now be even greater.
HIV health and social service settings provide a critical yet under-utilized opportunity to address these adverse impacts [7][8][9][10].According to a review by Hitch and colleagues (2019) [11], SUD screening within many HIV settings is limited.In addition to highlighting the need for HIV settings to conduct systematic SUD screening using validated tools, they noted screening must be accompanied by strategies to facilitate delivery of brief interventions and referral to SUD treatment [11].This recommendation is consistent with other calls to improve the integration of SUD services within HIV settings [7][8][9][10].More recently, as part of its Overdose Prevention Strategy, the Department of Health and Human Services appropriated $43.7 million dollars of funding in 2022 to support provision of accessible, evidence-based, and culturally appropriate SUD services for people with HIV or at high risk for HIV [12], and further HIV-focused SUD care allocations can be anticipated.The current implementation research, which focused on testing the incremental effectiveness of a pay-forperformance (P4P) strategy to help improve the implementation of an empirically-supported motivational interviewing brief intervention (MIBI) for SUD within HIV settings [13], may help inform how such funding is utilized.The current research may also help inform the eld of implementation science, which continues to need effective and cost-effective strategies for improving the implementation of evidence-based innovations within real-world practice settings.
The Substance Abuse Treatment to HIV Care (SAT2HIV) Project In 2014, as part of its effort to address comorbid HIV and SUD, the National Institute on Drug Abuse funded the Substance Abuse Treatment to HIV Care (SAT2HIV) Project.Using a dual-randomized type 2 hybrid trial design, 39 HIV service organizations (HSOs) were randomized to one of two implementation conditions [14] and within each HSO, client participants were randomized to one of two intervention conditions [15].The control implementation condition, based on research by Miller and colleagues (2004) [16], was a staff-focused strategy that provided 2 staff per HSO with training, performance feedback, and on-going consultation in implementing the MIBI with client participants.This is the strategy used by the Addiction Technology Transfer Center (ATTC), an intermediary organization funded by the Center for Substance Abuse Treatment, to prepare staff for implementing motivational interviewing.In addition to this strategy, a team-focused strategy called Implementation and Sustainment Facilitation (ISF) was provided to HSOs and their participating staff (2 MIBI staff and 1-3 leadership staff) randomized to the experimental condition.Grounded in the theory of implementation effectiveness [17][18][19], which posits implementation climate (i.e., the extent to which implementation is expected, supported, and rewarded) is a key mechanism of change, the ISF strategy focused on optimizing implementation climate during monthly 30-60 minute meetings with the HSO's MIBI staff and Leadership staff.These meetings were facilitated virtually by an ISF facilitator.Consistent with the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework [20,21], the focus of ISF meetings varied by EPIS phase.All ISF meetings, however, were guided by the principles of motivational interviewing: engaging, focusing, evoking, and planning [22][23][24].
The control intervention condition was usual care only (i.e., passive referral [e.g., pamphlet] to a local treatment provider), and the experimental intervention condition consisted of usual care plus a single 15 to 30-minute MIBI.As detailed by Garner and colleagues (2020) (11), compared to the staff-focused training, feedback, and consultation (TFC) control strategy, the addition of the ISF Strategy (TFC + ISF) resulted in signi cantly greater levels of implementation; both a greater number of MIBIs being implemented and a greater level of MIBI quality (i.e., delity) scores.Moreover, compared to usual care only, usual care plus the MIBI signi cantly reduced daily use of their primary substance for clients within the TFC + ISF condition but not within the TFC only condition [13].This evidence supporting the effectiveness of the MIBI is consistent with the broader body of evidence supporting the effectiveness of motivational interviewing [25][26][27].
Rationale for Testing Pay-For-Performance P4P is a strategy recommended to improve the quality and value of healthcare and to reduce healthcare disparities [28][29][30].P4P is a form of Powell and colleagues' (2015) [31] "alter incentive/allowance structures" strategy, which they de ned as "work to incentivize the adoption and implementation of the clinical innovation."Multiple meta-analytic studies support the effectiveness of using P4P to improve job performance [32][33][34][35].However, systematic reviews on the use of P4P within healthcare have concluded there remains insu cient rigorous evidence to make any strong conclusions [36][37][38].Although limited, there is some rigorous research demonstrating P4P can be an effective and cost-effective strategy for improving the implementation of evidence-based SUD interventions [39][40][41].According to a systematic review focused on quality improvement, implementation, and dissemination strategies to improve mental health care for children and adolescents [42], the P4P research conducted by Garner and colleagues (2012) provided the "strongest evidence in the review."Also supporting the rationale for testing P4P is that it is a strategy that directly addresses the reward dimension of implementation climate -a key mechanism of change according to the theory of implementation effectiveness [17][18][19].

Rationale for Trial Design
The current study used a cluster-randomized design (staff clustered within HSO [the unit of randomization]) to minimize the likelihood of contamination across the project's two implementation conditions, and because the cluster-randomized design may be preferable to alternatives, including the stepped-wedge trial design [43][44][45][46].In addition to the use of a cluster-randomized design, a type 3 hybrid trial was used because of the importance of examining both implementation outcomes and client outcomes [47][48][49][50][51].

Aims and Hypotheses
As illustrated in Fig. 1, the aims of this 26-site cluster-randomized type 3 hybrid trial included testing the effectiveness of P4P for improving implementation outcomes as well as client outcomes.We hypothesized that relative to MIBI staff trained and supported using the TFC + ISF strategy (control condition), which had been the most effective and costeffective strategy in the prior trial [13,52], that MIBI staff trained, supported, and rewarded using the TFC + ISF + P4P strategy (experimental condition) would achieve signi cantly greater: (a) MIBI implementation consistency, (b) MIBI implementation quality, (c) MIBI implementation effectiveness (i.e., the consistency and quality of implementation), and client participants in the experimental condition would achieve signi cantly greater (d) reductions in days of primary substance use, and (e) reductions in anxiety symptom severity.

Trial Design
The design was a 28-site cluster-randomized, type 3 implementation-effectiveness hybrid trial [48].Guided by the EPIS framework [20] HSOs (and their staff) were recruited and randomized to one of two implementation strategy conditions (TFC + ISF or TFC + ISF + P4P) during the project's exploration phase.During the project's preparation-implementation phase the TFC + ISF strategy was available to HSOs (and their staff) in both conditions.The P4P strategy was only available during the implementation phase and only to MIBI staff cluster-randomized to the TFC + ISF + P4P condition.The HSO was considered to have transitioned from the preparation phase to the implementation phase once a MIBI staff implemented a MIBI with a client participant.The preparation-implementation phase was 12-months for the project's rst cohort and 6-months for the project's second cohort.This difference was not planned, but was necessitated due to the project being temporarily stopped when the project's principal investigator changed institutions.Approval and oversight of all research activities was provided by the Institutional Review Board of Advarra.

Context
The project's participating HSOs (N = 26) were located in 13 states and the District of Columbia within the United States.

Participants
Staff participants.For an HSO to be eligible for the project, it had to receive funding from the Health Resources & Services Administrations' Ryan White program, have two or more staff members (e.g., case manager, behavioral specialist) willing to be trained to implement a MIBI for SUD, and have at least one leadership staff (e.g., director, manager, supervisor) willing to help ensure the MIBI staff were provided su cient time for participation.The participating HSOs identi ed individual staff to be invited to participate.Staff were required to be at least 18 years of age.There were not any exclusion criteria for HSO or staff participation.After staff provided informed consent to participate in the project, which was obtained electronically, they completed an online survey that collected background characteristics (e.g., age, biological sex, ethnicity, race, highest level of education).Participants received a US$25 e-gift card upon completion.
Client participants.To be eligible to participate, clients had to be at least 18 years of age and HIV positive.There were no exclusion criteria for client participation.Client participants were recommended to receive a MIBI when they reported unhealthy alcohol use or endorsed two or more of the 11 Diagnostic and Statistical Manual of Mental Disorders (DSM-V) criteria [53] for at least one substance during the past 12 months.Client participants received a US$10 e-gift card upon completion of the SUD screener.FMIBI-recommended client participants were asked to complete additional online assessments.These included a pre-MIBI assessment completed within the 24-hour period prior to receiving the MIBI and a follow-up assessment completed 4 weeks after receiving the MIBI.The MIBI was provided at no cost.Participants received e-gift cards upon completion of each assessment (US$20 for pre-MIBI assessment; US$10 for 4-week follow-up assessment).For the 4-week follow-up assessment, participants received a US$10 bonus for completion within 24 hours of the online link being emailed or texted to them or a US$5 bonus for completion within 25-48 hours of the online link being emailed or texted to them.

Strategies
Training, feedback, and consultation.The TFC strategy, similar to the ATTC strategy in the SAT2HIV Project [13,14], is a staff-focused strategy composed of three discrete strategies.As part of the current project, MIBI staff were provided Feedback on the quality of MIBI implementation was provided via a machine-learning based feedback system for motivational interviewing [54].For each MIBI session at least 10 minutes in length, a MIBI quality score was recorded.The range for each MIBI quality score was 0 to 12, with higher scores indicative of higher quality.MIBI staff were required to demonstrate MIBI pro ciency (a score of 4 or higher) via a practice role play session.MIBI role play sessions were conducted with another MIBI staff at their HSO who played the role of the client using a standardized client scenario provided by the project.
In addition to training and feedback, MIBI staff were provided the opportunity to participate in group MIBI consultation meetings.Conducted separately for each condition, these meetings were provided monthly, were 30-60 minutes in length, and were conducted by the trainer who conducted the virtual training sessions.In addition to providing didactic booster training regarding motivational interviewing and our project's MIBI protocol, these group consultations provided MIBI staff with the opportunity to ask speci c questions about how to improve the quality of MIBI implementation with client participants.Each consultation meeting was recorded and a link to the recording was emailed to the MIBI staff in the respective condition.
Implementation and sustainment facilitation.ISF is a team-focused strategy that seeks to optimize MIBI consistency and quality via optimizing the extent to which MIBI staff perceive these dimensions of implementation effectiveness to be expected, supported, and/or rewarded [13,14].It is team-focused due to requirement of engagement of staff to deliver the intervention as well as leaders to help address implementation barriers (e.g., competing priorities, insu cient time).To help optimize staff understanding of the implementation initiative and motivation for the training provided as part of the preparation phase, it is recommended that the rst ISF strategy meeting be completed prior to the initiation of the stafffocused training strategy.Consistent with other research that has used motivational interviewing as an implementation strategy [55,56], ISF uses key principles of motivational interviewing (i.e., engaging, focusing, evoking, planning) as a guide.Each monthly ISF strategy meeting was conducted virtually with only one HSO, lasted 30-60 minutes in length, and included three or more of the four motivational interviewing principles.An ISF workbook programmed in Microsoft Excel helped standardize the ISF strategy, document HSO staff attendance, document which motivational interviewing principles were employed, and document any meeting notes and/or action items.The ISF strategy was provided to each HSO by one of four master-level facilitators, two of which had provided ISF as part of the SAT2HIV Project.Weekly ISF team meetings supervised by a PhD-level ISF strategy facilitator were conducted for quality assurance purposes.ISF meetings were recorded and a link to the meeting recording was sent to the HSO's MIBI staff and leadership staff following each meeting.
Pay-for-Performance. P4P is a staff-focused strategy that utilizes nancial incentives and explicitly targets the reward dimension of implementation climate -the extent to which implementation is expected, supported, and rewarded [17][18][19].For each MIBI session implemented, the MIBI staff received a US$10 incentive.Additionally, for each MIBI session at least 10 minutes in length and that received a 6 + quality rating via a machine-learning based feedback system for motivational interviewing [54], the MIBI staff received a US$10 incentive.This machine-learning based system enabled MIBI staff to receive immediate feedback on session quality, including whether or not their overall score met or exceeded the project's pre-de ned quality target.The project's target was a score of 6, which was the 80th percentile score obtained by MIBI staff as part of SAT2HIV Project [13].
For the project's rst cohort, P4P incentives that incentivized attendance at the group consultation meetings and ISF meetings were introduced after month 6.Approved by the project's IRB, MIBI staff were asked to complete a new informed consent form to document their consent to receive a US$30 incentive per monthly group consultation meeting attended and a US$30 incentive per ISF meeting attended.Also approved by the IRB, leadership staff were asked to complete a new informed consent form that documented their consent to receive a US$30 incentive per ISF meeting attended.These additional P4P incentives were only for the HSOs and their participating staff that were randomized to the TFC + ISF + P4P condition.This revised P4P protocol was offered from the start for the project's second cohort.During the rst week of each month, research staff calculated the incentive amount earned by MIBI staff during the prior calendar month, sent a noti cation email to the staff, and had an e-gift card for the total incentive amount earned sent to the MIBI staff.There was not a limit on the amount MIBI staff were able to earn.

Intervention
The project's single session MIBI protocol has been shown to be effective in multiple settings [13,57].It was designed to help motivate an individual with comorbid HIV and SUD to change their primary substance use by: (a) examining their reasons for change, (b) receiving feedback about some common negative interactions between substance use and HIVrelated physical and mental health issues, (c) increasing the importance or con dence to reduce or stop their primary substance use, and (d) making a plan for change [15].

Outcomes
Implementation outcomes.Implementation effectiveness is a construct that others have operationally de ned as the consistency, which is similar to measures like reach [58] and penetration [59], and quality, which is similar to measures like delity [59] and integrity [60], of innovation implementation by targeted users [17][18][19].We de ned MIBI consistency as the total number of MIBIs a trained MIBI staff implemented during the project, with no minimum session length.We de ned MIBI quality as the total quality score a trained MIBI staff demonstrated during the project.For a MIBI quality score to be generated a MIBI session had to be at least 10 minutes in length.Each MIBI at least 10 minutes in length received a quality score between 0 and 12, with higher scores indicative of higher quality.Following the standardization of each respective measure, these two measures were summed together and standardized to create a measure of implementation effectiveness (i.e., the consistency and quality of implementation) [17][18][19].
Client outcomes.Consistent with its type 3 hybrid trial design, this project also examined the incremental impact of P4P on client outcomes 4-weeks post-MIBI.The two client outcomes of interest were change in days of primary substance use and change in anxiety symptom severity, both of which were based on client participant's self-report.They were asked "out of the past 28 days (4 weeks), about how many days did you use [insert their primary substance]."The Generalized Anxiety Disorder 7-item (GAD-7) [61], which ranges from 0 to 21 and where higher scores mean greater severity, was used to assess anxiety during the past two weeks.These client outcomes were aggregated for each MIBI staff.Thus, for each trained MIBI staff we computed two client outcome impact measures.One for their impact on days of primary substance use, and one for their impact on anxiety.A change score was computed for each possible client participant (i.e., those who completed the 4-week follow-up assessment) by subtracting their respective pre-MIBI assessment measure from their respective MIBI follow-up assessment measure.Then, for each respective change score measure we aggregated them to the level of the corresponding MIBI staff.Negative values are preferable as they represent greater reductions in the outcome.If a MIBI staff member was trained but implemented the MIBI with zero client participants, their client impact scores were zero (i.e., no impact, no return on investment).

Targeted sample size
The project proposed a cluster-randomized trial with 30 HSOs and 90 MIBI staff.Given the project was focused on testing the incremental impact P4P to improve implementation of a MIBI for SUD within HSO there was not a target number of client participants speci ed, nor was there a minimum number of client participants required per HSO or MIBI staff.

Randomization sequence generation
For each project cohort, randomization of HSOs (the clusters) was completed by the project's principal investigator, project coordinator, and statistician.The principal investigator and project coordinator independently matched pairs of HSOs based on information collected as part of an organizational background form.They then met to compare rankings and reach consensus on a nal list of match pairs.The nal list of matched pairs was given to the statistician who randomized each matched pair to one of the two implementation conditions.

Participant ow and recruitment
Two cohorts of HSOs were recruited and randomized as part of the project.The rst cohort, which cluster-randomized 12 HSOs, 44 MIBI staff, and 198 client participants, occurred from January 2021 through December 2021.The second cohort, which cluster-randomized 14 HSOs, 43 MIBI staff, and 143 client participants, occurred from November 2021 to April 2022.Figure 2 details the ow of the project's 26 HSO (87% of targeted sample), 87 MIBI staff (97% of targeted sample), and 341 client participants.As noted previously in the methods section, neither a target number or minimum number of client participants was pre-speci ed for the HSO or MIBI staff.
Half of the 26 HSOs were randomized to TFC + ISF and the other half were randomized to TFC + ISF + P4P.In the TFC + ISF condition, one HSO withdrew from the project after randomization and before any of the project's preparation phase strategies (ISF meetings, MIBI training) could be provided.The number of MIBI staff per condition was nearly the same, with 44 in TFC + ISF and 43 in TFC + ISF + P4P.The number of client participants in the TFC + ISF + P4P condition (n = 224) was 91% more than in the TFC + ISF condition (n = 117).The positive screening rate was 14 percentage points higher in TFC + ISF + P4P (80%) than in TFC + ISF (66%).There was little difference between conditions in terms of the number of client participants who withdrew from the study prior to completing the pre-MIBI assessment, with this rate being low for both conditions (10% for TFC + ISF and 8% for TFC + ISF + P4P).In addition to the SUD screening assessment, client participants were asked to complete a pre-MIBI assessment within the 24-hour period prior to receiving the MIBI.If the pre-MIBI assessment and MIBI were not completed within 30 days of the SUD screen, the MIBI participant was documented as lost.The percentage of client participants lost between completion of the SUD screener and the pre-MIBI assessment was 18 percentage points lower in TFC + ISF + P4P (30%; 50 of 165) than in TFC + ISF (48%; 33 of 69).If using the number of eligible client participants as the denominator, the percentage who received the MIBI was 13 percentage points higher in the TFC + ISF + P4P condition (58%; 104 of 180) than in the TFC + ISF condition (45%; 35 of 77).However, if using the number of client participants who were eligible and completed the pre-MIBI assessment, both conditions had excellent rates: 97% (35 of 36) in the TFC + ISF condition and 90% (104 of 115) in the TFC + ISF + P4P condition.Rates of completion for the 4-week post-MIBI assessment were nearly identical between conditions: 66% (23 of 35) for the TFC + ISF condition and 67% (70 of 104) for the TFC + ISF + P4P condition.

Blinding
There was no blinding of HSOs or their staff participants to implementation condition.Client participants were blind to implementation condition.

Statistical methods
Statistical analyses were conducted using an intention-to-implement approach.All MIBI staff who received some component of the TFC strategy or ISF strategy were included in the analyses.All analyses were conducted using SAS version 9.4 [62].An initial analytic step was to examine the partitioning of variance for each outcome measure.Mixed effects regression analyses were used for the impact of P4P on implementation outcomes and general linear regression analyses were used for the impact of P4P on client outcomes.Doubly robust estimation (i.e., combined propensity score weighting with outcome regression) was used to increase the models robustness to misspeci cation [63][64][65][66].Factors used to create the staff propensity weight were age, gender, ethnicity, race, education, and experience, both with the organization and in their current position with the organization.All analyses controlled for project cohort.We examined the extent to which there were any signi cant interactions between condition assignment and the staff factors included in the model.If no signi cant interaction was found, we focused on the main effect of the implementation condition.
Table 1 presents the baseline characteristics for MIBI staff participants for the overall sample (N = 87) and each condition (43 for TFC + ISF; 44 for TFC + ISF + P4P).Most MIBI staff were female (63%), non-Hispanic (77%), non-White (63%), and did not have a graduate degree (72%).On average, MIBI staff were 40.0 (SD = 12.5) years of age, had 51.4 (SD = 67.6)months of experience with their HSO, and 37.3 months (SD = 53.1) of experience in their current position.Table 2 presents the characteristics of client participants who were recommended for and received the project's MIBI overall (N = 139).See additional les for the characteristics of all client participants who consented to participate in the study and completed the SUD screener (N = 341) and the subset who were recommended for a MIBI (N = 257).
Consistent with a systematic review highlighting the need for studies that "move beyond discrete training events and towards longer term coaching-type activities focused on implementation" [70] Garner and colleagues (2020) [13] found adding the ISF strategy (a team-focused coaching-type strategy) on top of the TFC strategy (a staff-focused training-type strategy) signi cantly improved both (a) the implementation of a MIBI for SUD within HSOs, and (b) the effectiveness of the MIBI in reducing substance use.Building on this research, the aim of the current 26-site cluster-randomized type 3 hybrid trial was to test the incremental effectiveness of P4P (TFC + ISF vs. TFC + ISF + P4P) on both implementation outcomes and client outcomes.
In the current study, addition of P4P had a large and statistically signi cant impact on MIBI consistency (the number of MIBIs implemented), with the TFC + ISF + P4P condition implementing more than triple the number implemented within the TFC + ISF condition: 115 vs. 36.This large effect of P4P on implementation is consistent with prior research that found P4P to improve implementation of an adolescent substance use intervention [40].Although not an aim of the current project, future research examining the incremental cost-effectiveness of this P4P strategy may be warranted given how cost-effective the P4P was found to be in the trial with adolescents; a 5% increase in cost led to a 325% increase in clients receiving the pre-de ned dosage of treatment [41].
The addition of the P4P strategy also had a large effect on the overall level of MIBI quality achieved by MIBI staff, but this effect was not statistically signi cant according to our pre-speci c criteria of p < .05.Rather, in contrast to the statistically signi cant effect found for MIBI consistency, the impact of P4P on MIBI quality was slightly smaller and short of achieving statistical signi cance.This was also the case for MIBI implementation.Our nding that P4P had a slightly less positive effect on quality than on quantity is in line with ndings from a recent meta-analysis by Kim and colleagues (2022) [35].More speci cally, the meta-analysis by Kim et al. found that "the effect of incentives on performance was consistently and meaningfully larger for performance quantity than for quality," which is a nding they highlighted as being consistent with cognitive evaluation theory [71][72][73] and its later incorporation into self-determination theory [74][75][76][77].
Regarding impact on client outcomes, MIBI staff in the TFC + ISF + P4P condition had slightly greater overall changes/reductions in days of primary substance use by clients compared to the TFC + ISF condition (d = − .34),but this was not at all close to being statistically signi cant (p = .59).Although also not statistically signi cance (p = .11),we did nd MIBI staff who had more experience with their organization had greater overall changes/reductions in days of primary substance use by clients.However, given that more experience working in the HSO was associated with poorer MIBI quality (though not statistically signi cant), we believe it may be possible that MIBI staff with greater experience working in their HSO were able to establish stronger therapeutic rapport/alliance with their clients; a relationship that much research has found to be one of the best predictors of improved client outcomes.Although beyond the scope of the current research, it may therefore be warranted for future research to rate MIBI sessions for therapeutic rapport/alliance and to examine its relationship with client changes/reductions in days of substance use.
Despite not being the focus of the MIBI sessions, we did nd that client participants in the TFC + ISF + P4P condition had signi cantly greater changes/reductions in anxiety symptom severity (d = -1.54;p = .04)compared to those in the TFC + ISF condition.Although this nding is encouraging, it must be acknowledged that the more signi cant predictors of this outcome was if the MIBI staff was female (p = .03)or Hispanic (p = .007).There is research that has previously suggested there being a "female effect" [78], but we are not aware of research that has found similar effect for interventionists who identify as Hispanic.Future research exploring this issue may therefore be warranted.

limitations
The current research was not without limitations.First, while both implementation conditions were affected, conducting the research during COVID-19 was not ideal and may have limited the extent to which the MIBI was able to be implemented.Indeed, although conducting the MIBI via phone or virtually was also an option as part of the earlier trial [13], more MIBIs had to be conducted this way due to COVID-19 and may have limited the quality of MIBI implementation.
Second, client outcomes were limited to two measures that were based on self-report.Third, only about two-thirds of follow-up assessments were completed, which was lower than the 70% rate we had targeted as a minimum follow-up rate, and therefore does adversely impact generalizability of client-level ndings.

Conclusions and recommendations
The consistency and quality of implementation (i.e., implementation effectiveness) is believed to be the greatest when it is expected, supported, and rewarded [17][18][19].Building upon our prior research [13,40,41,52], the current study ndings support the effectiveness of using P4P to improve implementation of a MIBI for SUD within HIV service settings.An examination of cost-effectiveness is needed, as is an examination of how effective and cost-effective P4P may be for improving the implementation of other clinical innovations that are both evidence-based and underutilized in real-world practice settings.
access to the project's MIBI manual, 4-hour online introduction to motivational interviewing training course, and 12-hour virtual training in the MIBI protocol.Although conducted by the same trainer as the SAT2HIV Project, who was and remains part of the Motivational Interviewing Network of Trainers, the 12-hour virtual training was a condensed version of the 2-day in-person training used in the prior trial.This approach was necessitated due to COVID-19 safety protocols.The training content provided as part of the project's in-person training was condensed into three 4-hour virtual training sessions and was conducted over a four-week period.A one-week break between the second and third training sessions was provided to enable MIBI staff with time to practice the MIBI before the third training session.Free continuing education credits were provided for completion of both the online training and virtual training.
) 4.5 (0.6) Note: TFC = Training, Feedback, and Consultation; ISF = Implementation & Sustainment Facilitation; MIBI = motivational interviewing-based brief intervention, P4P = Pay For PerformanceTable 2 Client demographic characteristics (Received MIBI and Completed Follow-up Interview) = Training, Feedback, and Consultation; ISF = Implementation & Sustainment Facilitation; MIBI = motivational interviewing-based brief intervention, P4P = Pay For Performance AbbreviationsATTC Addiction Technology Transfer Center CI Con dence Interval DSM-V Diagnostic and Statistical Manual of Mental Disorders EPIS Exploration, Preparation, Implementation, Sustainment GAD-7 Generalized Anxiety Disorder 7-item HIV Human Immunode ciency Virus HSO HIV service organization ISF Implementation and Sustainment Facilitation MIBI Motivational Interviewing Brief Intervention P4P Pay for Performance PWH

Figures
Figures

Table 4
Results of analyses of the impact of the P4P strategy on implementation outcomes